Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tensor: Standardize Vector2, Vector3, Vector4, Color3, Color4, Quaternion, and Matrix #14235

Merged
merged 74 commits into from
Mar 5, 2024

Conversation

james-pre
Copy link
Contributor

@james-pre james-pre commented Aug 30, 2023

Discussion of this PR is available on the Babylon.js forum in this thread.
Resolves #14201

TL;DR:

Tensor would be added, which provides a standardized API for interacting with vectors, colors, matrices, quaternions, etc.

Rational

This PR aims to create a standard interface from which various math constructs are defined.

Take the current difference in behavior between vectors and colors:

const c1 = Color3.Random(),
	c2 = Color3.Random(),
	c3 = new Color3(),
	v1 = Vector3.Random(),
	v2 = Vector3.Random(),
	v3 = new Vector3();

c1.addToRef(c2, c3) === c3 // false
v1.addToRef(v2, v3) === v3 // true

And the lack of a method that should exist in Color3:

const v1 = Vector3.Random(),
	v2 = Vector3.Random(),
	c1 = Color3.Random(),
	c2 = Color3.Random();


v1.addInPlace(v2) // ok
c1.addInPlace(c2) // ReferenceError! addInPlace does not exist

Tensor provides a single interface for tensor-like objects which users can depend on. By having tensor-like classes implement Tensor, it ensures that all the methods exist and have the same and correct behavior. It is meant for organization and standardization.

Semantics

Note:

  • CurrentClass is a placeholder for a class that implements Tensor. In fields, it is the current class.
  • Tensor and its related types will be defined in src/Maths/tensor.ts unless otherwise noted
  • Signatures are shown, not implementation.
  • The shown signatures are not necessarily the same as implementation (e.g. Signature<T extends number> may be shown when it is implemented as Signature<T> = T extends number ? ... : never

Status

* May not be included in the proposal

Vector

Defined in src/Maths/math.vector.ts

This PR includes Vector from #13699. Vector extends Tensor and adds normalization methods, length, and lengthSquared. It is implemented by Vector2, Vector3, and Vector4.

Tuple manipulation and math types

Defined in src/types.ts
The following types are being added for compile-time math and tuple manipulation, which is necessary for MultidimensionalArray. They are included as a single block for brevity.

type Empty = [];
type Shift<T extends unknown[]>;
type First<T extends unknown[]>;
type Unshift<T extends unknown[], A>;
type Pop<T extends unknown[]>;
type Last<T extends unknown[]>;
type Push<T extends unknown[], A>;
type Concat<A extends unknown[], B extends unknown[]>;
type Remove<A extends unknown[], B extends unknown[]>;
type Length<T extends unknown[]>;
type FromLength<N extends number>;
type Increment<N extends number>;
type Decrement<N extends number>;
type Add<A extends number, B extends number>;
type Subtract<A extends number, B extends number>;
type Member<T, D>;
type Flatten<A extends unknown[], D>;
type MultidimensionalArray<T, D extends number>
type Tuple<T, N extends number>;
type FlattenTuple<T>;
type IsTuple<T>;

Constructor

Defined in src/types.ts

type Constructor<C extends new (...args: any[]) => any, I extends InstanceType<C>>

Defines a single type for constructors. This replaces Vector2Constructor, Vector3Constructor, Vector4Constructor, QuaternionConstructor, and MatrixConstructor.

Instead of using Vector3Constructor<this>, you would instead use Constructor<typeof Vector3, this>. The typeof is needed to get the type for the class instead of the instance, and can't be done inside the type since Typescript prohibits taking the typeof a type.

MultidimensionalArray

Defined in src/types.ts

type MultidimensionalArray<T, D extends number>

Represents a multidimensional array of T with depth D.

TensorValue

type TensorValue<T> = T extends Tensor<infer V> ? V : never;

Tensor

declare class Tensor<V>

Tensor is a declare class for tensor-like classes to implement.

Tensor includes all of the methods in Vector from #13699. This includes

  • Math operations (add, subtract, multiply, divide, scale, ...)
  • Array conversion (fromArray, toArray, asArray, ...)
    • Since the type parameter is no longer assignable to number[], the return type for array-related methods is number[]. See Tensor.value
  • Transferring (clone, copyFrom, copyFromFloats, ...)
  • The above methods' InPlace, ToRef, FromFloats, etc.

It adds or changes the following methods:

Tensor.dimension

public abstract dimension: number[];

dimension is the mathematical dimension2 of the tensor. In dynamic tensor types, it can be defined as a getter.

Example:

const vec3 = new Vector3(),
	vec4 = new Vector4(),
	matrix = new Matrix();

vec3.dimension // [3]
vec4.dimension // [4]
matrix.dimension // [4, 4]

Tensor.From

public static From(source: Tensor, fillValue: number = 0): CurrentClass;

From creates a new instance of CurrentClass from source.
source: The tensor to copy data from.
fillValue: The value to use for filling empty parts of the resulting CurrentClass.

Example:

const vec3 = new Vector3(1, 2, 3);
const vec4 = Vector4.From(vec3, 4); // { 1, 2, 3, 4 }
const matrix = Matrix.From(vec3, 0); // this is possible!

Additionally, From could be changed to From(...args: [...Tensor[], number]): CurrentClass to allow multiple inputs. This is better understood by this invalid Typescript signature (since rest parameters must be last):

public static From(...source: Tensor[], fillValue: number = 0): CurrentClass;

🛈 Implementations may write their own conversion code or use convertTensor.

Tensor.as

public as<T extends typeof Tensor>(type: T, fillValue: number = 0): InstanceType<T>;

as creates an instance of type from an instance of CurrentClass.
type: The class to create an instance of.
fillValue: The value to use for filling empty parts of the resulting instance.

Example:

const vec3 = new Vector3(1, 2, 3);
const vec4 = vec3.as(Vector4, 4); // { 1, 2, 3, 4 }
const matrix = vec3.as(Matrix, 0);

🛈 Implementations may write their own conversion code or use convertTensor.

Tensor.sum

public sum(): number;

sum return the sum of the components of the Tensor.

Example:

const vec3 = new Vector3(1, 2, 3),
	vec4 = new Vector4(4, 5, 6, 7),
	matrix = new Matrix();

vec3.sum() // 6
vec4.sum() // 22
matrix.sum() // 0

⚠ The return value of the lengthSquared or length method of vectors is not the same as the return value of sum.

Tensor.rank

public abstract rank: number;

The rank of a tensor is the number of indices required to uniquely select each element of the tensor.

Example:

const vec3 = new Vector3(),
	matrix = new Matrix();

vec3.rank // 1
matrix.rank // 2

🛈 The rank of a Tensor is the same as its dimension.length.

Tensor.value

public get value(): V;
public set value(value: V): void;

value is the values of the tensor in a multidimensional array with the type V (the type parameter of the class). Unlike dimension, this must be implemented as a getter and setter.

Example:

const vec3 = new Vector3(),
	matrix = new Matrix();

vec3.value
// [0, 0, 0]

matrix.value
// [ [ 0, 0, 0, 0 ], [ 0, 0, 0, 0 ], [ 0, 0, 0, 0 ], [ 0, 0, 0, 0 ] ]

vec3.value = [ 1, 2, 3 ];
vec3.x // 1

isTensor

function isTensor<V>(value: unknown): value is Tensor<V>;

Checks if a value is a Tensor.
Example:

const vec3 = new Vector3();
isTensor(vec3) // true
isTensor(vec3.value) // false
isTensor([1, 2, 3]) // false
isTensor(0) // false
isTensor(null) // false

isTensorValue

function isTensorValue<T>(value: unknown): value is TensorValue<T>;

Checks if a value is a TensorValue.
Example:

const vec3 = new Vector3();
isTensorValue<1>(vec3) // false
isTensorValue<Vector3>(vec3.value) // true
isTensorValue<1>([1, 2, 3]) // true
isTensorValue<0>([1, 2, 3]) // false
isTensorValue<1>(0) // false
isTensorValue<0>(0) // true
isTensorValue<1>(null) // false

getTensorDimension

function getTensorDimension(value: Tensor | TensorValue): number[];

getTensorDimension returns the mathimatical dimension2 of value, similar to Tensor.dimension. If value is not a Tensor or TensorValue, it will throw a TypeError

🛈 Dynamic Tensor based classes should use getTensorDimension for their dimension implementations. Fixed-dimension Tensor sub-classes need not use getTensorDimension.

(optional) convertTensor

function convertTensor<T extends  typeof Tensor>(TensorClass: T, tensor: Tensor, fillValue: number = 0): InstanceType<T>;

Creates a new T with the values of tensor. Unlike Tensor.as, it performs more error checking and does not need a starting instance (and therefore avoids the dreaded 'methodName' does not exist on undefined error).

Behavior changes

  • The Color3 and Color4 methods multiplyToRef, scaleToRef, scaleAndAddToRef, clampToRef, addToRef, and subtractToRef have had their return types changed from this to the reference instance.

Considerations

  1. Static methods related to data manipulation (e.g. Add, Normalize, Lerp) and non-static normalization methods are outside the scope of this PR. While Tensor may include them in the future, this PR does not
  2. This PR does not include a dimensionally dynamic Tensor.
  3. Performance: Since Tensor is defined using the declare class and classes that follow Tensor do so using the implements keyword, there are no runtime changes to Tensor-based classes.
  4. Bundle size: The JS bundle size will increase only by the size of the added functions (isTensor, getTensorDimension, etc.). The TS declaration size will also increase slightly due to Tensor and the added utility types.
  5. Maintenance: Since there is no implementation (just a class declaration), the extra maintenance is minimized to member signatures.

Questions

For the BabylonJS team concerning the PR

  1. What other classes (e.g. Size) would be included and standardized?

FAQs

What use cases does this proposal have?

Tensor is not intended to be used by users. It is meant for internal organization and standardization. The functions (isTensor, getTensorDimension, etc.) are intended to be public and may be used by users. Their use cases may be inferred from their descriptions.

What is the difference between this and #13699?

The capabilities of the type. Vector supports any rank 1 tensor. Tensor generalizes Vector, and supports any rank tensor. A mathematical vector can be represented as number[]. A mathematical tensor can be represented as a number, number[], number[][], and so on. This table highlights the capability:

tensor rank TS representation BJS class (if exists)
1 number[] Vector
1 [number, number] Vector2
1 [number, number, number] Vector3, Color3
1 [number, number, number, number] Vector4, Color4, Quaternion
2 number[][] Matrix
3 number[][][]
4 number[][][][]
N TensorValue

How is incompatibility managed? Are incompatible classes dropped or are they modified to follow Tensor?

Any classes included will be modified to follow the standard. In many cases, the diverging behavior makes the engine difficult to use, and changing the claseses to follow the Tensor standard benefits end users.

Further reading

1. https://en.wikipedia.org/wiki/Tensor
2. https://en.wikipedia.org/wiki/Dimension

@bjsplat
Copy link
Collaborator

bjsplat commented Aug 30, 2023

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

@bjsplat
Copy link
Collaborator

bjsplat commented Aug 30, 2023

@RaananW
Copy link
Member

RaananW commented Sep 4, 2023

Small note - there are a few type issues, building core was not successful.

@james-pre
Copy link
Contributor Author

@kzhsw Thank you so much for the reviews. Another set of eyes helps to catch things missed.

Updated MultidimensionalArray to correctly handle nesting
@jstroh
Copy link

jstroh commented Sep 13, 2023

I love it!

@GuoBinyong
Copy link
Contributor

When this plan will be realized, this design is useful

@sebavan sebavan mentioned this pull request Nov 10, 2023
@bjsplat
Copy link
Collaborator

bjsplat commented Feb 20, 2024

Visualization tests for WebGPU (Experimental)
Important - these might fail sporadically. This is an optional test.

https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14235/merge/testResults/webgpuplaywright/index.html

@RaananW
Copy link
Member

RaananW commented Feb 20, 2024

The only way I know how to manage static members with interfaces is this:

https://www.typescriptlang.org/play?#code/JYOwLgpgTgZghgYwgAgJIDUILAeygZTDjGAWQG8AoZG5ECAdwAoBKALjU2zwG5rkAwgBs4AWwAOrDhiy4ofAL6VKoSLEQoZ3KBX5wAJvqmdZvSksoIcIAM5hkXOQGZpjvIWKlkAXmQIRNjbIwBJCEKIQ4EFacro0VrZgUACu2qwUSjQGRuwm2nG0UBBgyVAgyGAAFsA2PMj8mch2nmTCYpK5MXgFNEUlZXSMDqZQTqx8NEoWCXbIAG4+gwzD2mMsfHMAdNnjlG6jm20S40A

I am not the biggest fan of this way, but this is the only way (I know :-)) to define static members for a class. @sebavan - we can discuss this offline and see whether or not this solves the issue.

@james-pre
Copy link
Contributor Author

@sebavan @RaananW,

I've separated the instance and static sides into interfaces for both Tensor and Vector, as described in (B) in my last message. I also mentioned that using interfaces comes with the benefit of allowing a type parameter on the static side, which is very useful.

I also looked into defining Tensor.constructor (instance property) as TensorStatic<this>, which would narrow the type of this.constructor in all Tensor subclass instance methods to the class (again, very useful). This does not work unfortunately as TS narrows the type of constructor to something not very useful, which is not compatible with the static side interface. This unfortunately means we need to keep the this.constructor as ... type assertions throughout the instance code.

I apologize for not responding sooner @RaananW, I was experimenting with the constructor property and types.

The only way I know how to manage static members with interfaces is this

The way I prefer, which also preserves the class syntax, is with the satisfies keyword, like in this example:

export class Vector2 implements Vector<Tuple<number, 2>> {
    // ...
}
Vector2 satisfies VectorStatic<Vector2>;

Please note that the left-hand side of satisfies is a value and the right-hand side is a type.

Added Vector2 and Vector3 RandomToRef
Removed DistanceOfPointFromSegment from Vector
Added static side checks to Vector2 and Vector3
@bjsplat
Copy link
Collaborator

bjsplat commented Feb 20, 2024

Visualization tests for WebGPU (Experimental)
Important - these might fail sporadically. This is an optional test.

https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14235/merge/testResults/webgpuplaywright/index.html

Moved Clamp and ClampToRef from Vector to Tensor
@sebavan
Copy link
Member

sebavan commented Feb 27, 2024

Let s use @RaananW proposal to do it instead of satisfies as it has only been introduced really late in TS and could create compat issues with the users I guess.

@james-pre
Copy link
Contributor Author

james-pre commented Feb 28, 2024

@sebavan,

Typescript < 4.9 is already not usable with BJS so using satisfies should be fine. When trying to npm run build:dev with TS ~4.8.0:

packages/dev/core/src/Engines/WebGPU/webgpuTextureManager.ts:1164:99 - error TS2304: Cannot find name 'VideoFrame'.

1164         imageBitmap: ImageBitmap | Uint8Array | ImageData | HTMLImageElement | HTMLVideoElement | VideoFrame | HTMLCanvasElement | OffscreenCanvas,
                                                                                                       ~~~~~~~~~~

packages/dev/core/src/Engines/WebGPU/webgpuTextureManager.ts:1275:106 - error TS2304: Cannot find name 'VideoFrame'.

1275             imageBitmap = imageBitmap as ImageBitmap | ImageData | HTMLImageElement | HTMLVideoElement | VideoFrame | HTMLCanvasElement | OffscreenCanvas;

@RaananW
Copy link
Member

RaananW commented Feb 28, 2024

@sebavan,

Typescript < 4.9 is already not usable with BJS so using satisfies should be fine. When trying to npm run build:dev with TS ~4.8.0:

packages/dev/core/src/Engines/WebGPU/webgpuTextureManager.ts:1164:99 - error TS2304: Cannot find name 'VideoFrame'.

1164         imageBitmap: ImageBitmap | Uint8Array | ImageData | HTMLImageElement | HTMLVideoElement | VideoFrame | HTMLCanvasElement | OffscreenCanvas,
                                                                                                       ~~~~~~~~~~

packages/dev/core/src/Engines/WebGPU/webgpuTextureManager.ts:1275:106 - error TS2304: Cannot find name 'VideoFrame'.

1275             imageBitmap = imageBitmap as ImageBitmap | ImageData | HTMLImageElement | HTMLVideoElement | VideoFrame | HTMLCanvasElement | OffscreenCanvas;

I am not a big fan of any of these ways. Each way of validating statics has its downside. the downside of satisfies is that it fails not as part of the class. It's an assertion line (i know you can claim any typing is basically assertion). So the error is shown during the assertion, and not during the class declaration.

The typescript version that Seb mentioned is not necessarily the version we use to compile the framework, it is the language that typescript developers can use. If not using WebGPU you would be able to continue using the same typescript version that you were using, if that's your decision. It is also possible to add the definition of the missing declaration. However, satisfies is a typescript keyword that doesn't allow it. We try our best not to do that.

@james-pre
Copy link
Contributor Author

Unfortunately using const Class: StaticSide = class { ... will not work since it is an assignment. This means that any members present on the static side of the class will not be known by TS in the Class variable, since it is declared as StaticSide. For example:

interface StaticSide {
    DoSomething(): void;
}

const Class: StaticSide = class {
    DoSomething() {
        // ...
    }
    
    OtherThing() {
        // ...
    }
}

Class.OtherThing(); // <-- Error! OtherThing does not exist on StaticSide

This can be solved by using a separate variable though:

interface StaticSide { /* ... */ }
export class Class { /* ... */ }

/*
This does not change the type of Class.
It is also not exported
Because declare is used this will not exist at runtime
*/
declare const $typecheck$Class: StaticSide = Class;
```

@RaananW
Copy link
Member

RaananW commented Feb 29, 2024

Unfortunately using const Class: StaticSide = class { ... will not work since it is an assignment. This means that any members present on the static side of the class will not be known by TS in the Class variable, since it is declared as StaticSide. For example:

interface StaticSide {
    DoSomething(): void;
}

const Class: StaticSide = class {
    DoSomething() {
        // ...
    }
    
    OtherThing() {
        // ...
    }
}

Class.OtherThing(); // <-- Error! OtherThing does not exist on StaticSide

This can be solved by using a separate variable though:

interface StaticSide { /* ... */ }
export class Class { /* ... */ }

/*
This does not change the type of Class.
It is also not exported
Because declare is used this will not exist at runtime
*/
declare const $typecheck$Class: StaticSide = Class;

have you seen this -

https://www.typescriptlang.org/play?#code/JYOwLgpgTgZghgYwgAgJIDUILAeygZTDjGAWQG8AoZG5ECAdwAoBKALjU2zwG5rkAwgBs4AWwAOrDhiy4ofAL6VKoSLEQoZ3KBX5wAJvqmdZvSksoIcIAM5hkXOQGZpjvIWKlkAXmQIRNjbIwBJCEKIQ4EFacro0VrZgUACu2qwUSjQGRuwm2nG0UBBgyVAgyGAAFsA2PMj8mch2nmTCYpK5MXgFNEUlZXSMDqZQTqx8NEoWCXbIAG4+gwzD2mMsfHMAdNnjlG6jm20S40A

Of course, if you implement it the way you showed it wouldn't work :-)

Again - there is no good solution here. I am, in general, not in favor of any of those solutions. I have the least resistance to this solution.

@james-pre
Copy link
Contributor Author

@RaananW
Copy link
Member

RaananW commented Mar 1, 2024

I'll be honest - this discussion is way too deep for something we won't change for a long time :-)

Whatever fits. Satisfies as assertion feels a bit off, but typescript does support it. Other things are as hacky as it is. I am fine with satisfies (in THIS case) for the same reason as I wrote above - Vectors won't change any time soon.

@dr-vortex - I am getting to the last question on my side, and it is perf. Let's run a few more perf tests, sharing what we use to run them, to know 100% we don't hurt anything (and maybe even improve a bit). This won't be affected by any interface/satisfies changes.

@deltakosh
Copy link
Contributor

I'm for it! Let's merge it

@james-pre
Copy link
Contributor Author

@sebavan,

If you have no other concerns could you please approve the PR (it currently says you are requesting changes)? Thank you very much!

@sebavan
Copy link
Member

sebavan commented Mar 5, 2024

Waiting on @RaananW perf check so it prevents accidental merge

@RaananW
Copy link
Member

RaananW commented Mar 5, 2024

@sebavan - want to resolve your review? there are still open change requests

@sebavan
Copy link
Member

sebavan commented Mar 5, 2024

@RaananW approved but let us know your perf results

@RaananW
Copy link
Member

RaananW commented Mar 5, 2024

@RaananW approved but let us know your perf results

Every scene I have tested was on par with the current implementation. I tested heavy scenes (like the performance priority PG and the physics stress testing), and the PG i pasted before testing the function implementations directly (including creation and basic math).
There was no perf bottleneck from the vector implementation when profiling any of the scenes.
I am pressing the merge button :-)

@RaananW RaananW merged commit 9258609 into BabylonJS:master Mar 5, 2024
11 checks passed
@james-pre james-pre deleted the tensor branch March 5, 2024 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Standardize Scalar, Vector2, Vector3, Vector4, Color3, Color4, Quaternion, Matrix
9 participants